feat: Timeout handling and additional plugin pool improvements#691
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis PR restructures the plugin timeout architecture from a single pool-request timeout to a dual-tier model: per-plugin execution timeouts (via config.json, default 300s) and HTTP server request timeout, with corresponding updates to constants, error handling, and Rust-side backstop logic across both Node.js and Rust implementations. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Comment Tip CodeRabbit can use OpenGrep to find security vulnerabilities and bugs across 17+ programming languages.OpenGrep is compatible with Semgrep configurations. Add an |
Codecov Report❌ Patch coverage is Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more. @@ Coverage Diff @@
## main #691 +/- ##
========================================
Coverage 90.97% 90.97%
========================================
Files 288 288
Lines 118596 118698 +102
========================================
+ Hits 107888 107984 +96
- Misses 10708 10714 +6
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/plugins/index.mdx`:
- Around line 847-853: Update the docs to recommend a buffer so
REQUEST_TIMEOUT_SECONDS is strictly greater than your longest plugin timeout
(plugin.timeout) to avoid the race that causes write EPIPE; explicitly state
"REQUEST_TIMEOUT_SECONDS should be at least X seconds greater than the longest
plugin.timeout (e.g., +1–2s buffer)" and apply the same change to the other
occurrence referenced (around the lines noted).
In `@plugins/lib/worker-pool.ts`:
- Around line 107-108: The outer worker-pool uses a fixed DEFAULT_TASK_TIMEOUT
(DEFAULT_PLUGIN_TIMEOUT_MS + 5000) which ignores per-plugin request.timeout,
causing plugins with timeouts > DEFAULT_PLUGIN_TIMEOUT_MS to be cut off; update
the code that sets taskTimeout (the value passed into the Piscina safety timeout
/ Promise.race) to derive from the effective per-request timeout (use
request.timeout when present, otherwise DEFAULT_PLUGIN_TIMEOUT_MS), then add the
5000ms cleanup buffer and use that computed value instead of
DEFAULT_TASK_TIMEOUT when invoking Piscina.runTask / Promise.race so the pool
honors per-plugin timeouts.
In `@src/services/plugins/pool_executor.rs`:
- Around line 996-1000: The response_timeout calculation (using
response_timeout, timeout_secs, DEFAULT_PLUGIN_TIMEOUT_SECONDS) adds a fixed 5s
buffer but still can be shorter than the actual worker-side wait, causing
callers to get a generic timeout before the worker can return the structured
504/TIMEOUT; update the response_timeout computation to match or exceed the
worker wait by using the same buffer/delay logic the worker uses (replace the
hardcoded + Duration::from_secs(5) with the same buffer constant or value used
by the worker, or compute response_timeout =
(timeout_secs.unwrap_or(DEFAULT_PLUGIN_TIMEOUT_SECONDS) + WORKER_BUFFER_SECS +
QUEUE_BUFFER_SECS) seconds) and apply the same change to the other occurrence
referenced in the comment (the block around the second occurrence at the later
lines) so both code paths use the identical timeout math.
- Around line 1096-1118: The extra broad string check in is_dead_server_error is
preventing DeadServerIndicator::from_error_str from detecting transport-level
timeouts (e.g., PluginExecutionError("connection timed out")); remove the
(lower.contains("plugin") && lower.contains("timed out")) clause and only
special-case the explicit Node.js phrase "handler timed out" so that
connection/transport timeout messages fall through to
DeadServerIndicator::from_error_str for proper dead-server detection.
- Around line 545-549: The current code adds PLUGIN_TIMEOUT_BUFFER_SECONDS into
the timeout passed to conn.send_request_with_timeout, causing ScriptTimeout to
report the expanded wall-clock time; instead pass the original configured
timeout (timeout_secs.unwrap_or(DEFAULT_PLUGIN_TIMEOUT_SECONDS)) to
send_request_with_timeout so callers see the intended timeout, and implement the
backstop separately (e.g., run a parallel watchdog using
PLUGIN_TIMEOUT_BUFFER_SECONDS that forcibly terminates the Node process if the
plugin hangs) so the buffer is used only for enforcement and not reflected in
the reported PluginError; update the call site where timeout is computed and the
call conn.send_request_with_timeout(&request, timeout) to use the
configured_timeout and add a watchdog/select-based backstop surrounding that
call.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: aecb0fd2-2cee-4211-a843-896e8b3b89a7
📒 Files selected for processing (14)
docs/plugins/index.mdxplugins/ARCHITECTURE.mdplugins/lib/compiler.tsplugins/lib/constants.tsplugins/lib/plugin.tsplugins/lib/pool-executor.tsplugins/lib/pool-server.tsplugins/lib/worker-pool.tssrc/constants/plugins.rssrc/services/plugins/config.rssrc/services/plugins/connection.rssrc/services/plugins/mod.rssrc/services/plugins/pool_executor.rssrc/services/plugins/shared_socket.rs
💤 Files with no reviewable changes (1)
- plugins/ARCHITECTURE.md
There was a problem hiding this comment.
Pull request overview
This PR improves plugin pool timeout behavior and resiliency so that plugin execution timeouts return actionable, structured responses (HTTP 504 / TIMEOUT) and expected socket errors during timeout teardown don’t crash the pool server or spam logs.
Changes:
- Route
ScriptTimeoutthrough the normal handler path to return structured HTTP 504 responses (withcode: "TIMEOUT"). - Implement a layered timeout hierarchy (handler → worker-pool safety net → Rust backstop) with explicit buffer constants.
- Harden pool-server / socket write behavior during disconnects (EPIPE/ECONNRESET) and simplify timeout-related configuration.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/services/plugins/shared_socket.rs | Downgrades expected socket write errors to DEBUG via a helper and adds basic tests. |
| src/services/plugins/pool_executor.rs | Adds timeout buffering/backstop logic, replaces request-timeout config usage with constants, and refines dead-server detection. |
| src/services/plugins/mod.rs | Converts ScriptTimeout into a structured handler response (504, TIMEOUT) and adds test coverage. |
| src/services/plugins/connection.rs | Changes pool request timeout mapping to PluginError::ScriptTimeout. |
| src/services/plugins/config.rs | Removes pool_request_timeout_secs from config. |
| src/constants/plugins.rs | Introduces timeout-buffer/admin-timeout constants and removes old pool request timeout constants. |
| plugins/lib/worker-pool.ts | Uses per-request timeout to derive worker safety-net timeout (avoids cutting off long-timeout plugins). |
| plugins/lib/pool-server.ts | Adds safeWrite and ignores expected disconnect errors in global error handlers. |
| plugins/lib/pool-executor.ts | Renames socket request timeout constant usage. |
| plugins/lib/plugin.ts | Renames socket request timeout constant usage. |
| plugins/lib/constants.ts | Splits execution vs per-API-call socket timeouts; introduces DEFAULT_PLUGIN_TIMEOUT_MS. |
| plugins/lib/compiler.ts | Minor comment tweak for compile timeout. |
| plugins/ARCHITECTURE.md | Removes PLUGIN_POOL_REQUEST_TIMEOUT_SECS from docs table. |
| docs/plugins/index.mdx | Updates guidance to per-plugin timeout + HTTP request timeout alignment; removes PLUGIN_POOL_REQUEST_TIMEOUT_SECS. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
LGTM! just one question:
I have one concern here:
Could this change make some pool/server timeouts look like normal plugin timeouts? Since Rust now turns any SocketError with "timed out" into ScriptTimeout, I think a stuck Node pool might return a regular 504 timeout instead of going through the recovery/restart path.
(this is a finding found by Codex analysis :))
…nto plugin-pool-improvements
…nto plugin-pool-improvements
Thanks for feedback. Sai addressed it already. |
Summary
When a plugin exceeds its timeout, the API previously returned a generic HTTP 500 with "Internal server error". This PR makes timeouts actionable:
message
teardown and handled gracefully
preventing false-positive server restarts
Testing Process
Checklist
Note
If you are using Relayer in your stack, consider adding your team or organization to our list of Relayer Users in the Wild!
Summary by CodeRabbit
New Features
Documentation
Bug Fixes